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Abstract 

Background: Interoperable electronic health record (EHR) solutions are currently being implemented in Canada, as 
in many other countries. Understanding EHR users' perspectives is key to the success of EHR implementation 
projects. This Delphi study aimed to assess in the Canadian context the applicability, the importance, and the 
priority of pre-identified factors from a previous mixed-methods systematic review of international literature. 

Methods: A three-round Delphi study was held with representatives of 4 Canadian EHR user groups defined as 
partners of the implementation process who use or are expected to use EHR in their everyday activity. These 
groups are: non-physician healthcare professionals, health information professionals, managers, and physicians. Four 
bilingual online questionnaire versions were developed from factors identified by the systematic review. Participants 
were asked to rate the applicability and the importance of each factor. The main outcome measures were 
consensus and priority. Consensus was defined a priori as strong (> 75%) or moderate (> 60-74%) according to 
user groups' level of agreement on applicability and importance, partial (> 60%) when participants agreed only on 
applicability or importance, or as no consensus (< 60%). Priority for decision-making was defined as factors with 
strong consensus with scores of 4 or 5 on a five-point Likert scale for applicability and importance. 

Results: Three Delphi rounds were completed by 64 participants. Levels of consensus of 100%, 64%, 64%, and 44% 
were attained on factors submitted to non-physician healthcare professionals, health information professionals, 
managers, and physicians, respectively. While agreement between and within user groups varied, key factors were 
prioritized if they were classified as strong (> 75% from questionnaire answers of user groups), for decision-making 
concerning EHR implementation. The10 factors that were prioritized are perceived usefulness, productivity, 
motivation, participation of end-users in the implementation strategy, patient and health professional interaction, 
lack of time and workload, resources availability, management, outcome expectancy, and interoperability. 

Conclusions: Amongst all factors influencing EHR implementation identified in a previous systematic review, ten 
were prioritized through this Delphi study. The varying levels of agreement between and within user groups could 
mean that users' perspectives of each factor are complex and that each user group has unique professional 
priorities and roles in the EHR implementation process. As more EHR implementations in Canada are completed it 
will be possible to corroborate this preliminary result with a larger population of EHR users. 

Keywords: Delphi technique, Adoption factors, Implementation factors, Electronic health record, Health information 
technology, Health communication technology, Medical informatics 
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Background 

Electronic health records 

There is currently worldwide interest in the potential of 
electronic health record (EHR) to reduce healthcare 
costs and improve significantly the quality of healthcare 
provided [1]. EHR programs are perceived as an oppor- 
tunity to improve the health sector fundamentally. 
Nevertheless, these programs are complex and costly [2]. 
Deutsch and colleagues evaluated EHR programs in five 
countries (England, Germany, Canada, Denmark, and 
Australia) and found five critical areas for successful im- 
plementation: acceptance and change management, 
demonstration of benefits and funding, project manage- 
ment, health policy-related goals and implementation 
strategy [2]. 

EHRs differ from electronic medical records (EMRs) 
and personal health records (PHRs) most notably on the 
basis of the completeness of the information the record 
contains and the designated custodian of the information 
[3]. A PHR is often described as being a complete or par- 
tial health record under the custodianship of a person(s) 
(e.g. a patient or family member) that holds all or a por- 
tion of the relevant health information about that person 
over their lifetime. This is a person-centric health record. 
An EMR may be characterized as a partial health record 
under the custodianship of a health care provider(s) that 
holds a portion of the relevant health information about 
a person over their lifetime. This is often described as a 
provider-centric or health organization-centric health 
record of a person. In contrast, an EHR may be defined 
as a complete health record under the custodianship of a 
healthcare provider(s) that holds all relevant health infor- 
mation about a person over their lifetime. This is often 
described as a person-centric health record, which can 
be used by many approved healthcare providers or health 
care organizations. 

In Canada, a network of interoperable electronic 
health record (EHR) network is however currently being 
implemented. Canada Health Infoway was created by 
Canada's First Ministers in 2001 with the goal to foster 
and accelerate the building of a pan-Canadian electronic 
health record network which will manage Canadians' 
health information. According to Canada Health Info- 
way, EHR solutions will link clinics, hospitals, pharma- 
cies and other points of care. They hold the promise of 
helping to improve Canadians' access to health services, 
enhance the quality of care and patient safety, and help 
the health care system becoming more efficient [4]. 

Recent systematic reviews show that EMR/EHR sys- 
tems have not yet demonstrated clinical and economic 
benefits [1] [5] [6]. Other recent results indicate that 
computerized clinical decision support systems 
(CCDSSs), a feature that is often linked to the EHR, in- 
consistently improved process of care measures and 



seldom improved patient outcomes. Lack of clear patient 
benefit and lack of data on harms and costs precluded a 
recommendation to adopt CCDSSs for drug therapy 
management [7]. Moreover, it is possible that harms 
may neutralize many of the benefits expected from 
EMR, EHR or CCDSS [7]. The vision of Canada Health 
Infoway is to provide a secure and private EHR lifetime 
record of health history and care within the health sys- 
tem, available to authorized health providers and indivi- 
duals. It is believed that EHRs will facilitate the sharing 
of data across the continuum of care, the healthcare de- 
livery organizations, and geographical areas [8]. The 
Canadian Medical Association has called for government 
investment in health information systems such as EHRs 
with the goal of improving patient outcomes and system 
efficiency [9,10]. Nevertheless, the Canadian healthcare 
system remains paper-laden and EHR implementation 
lags behind many other industrial countries [11-14]. 

EHR implementation involves many user groups. In 
this study, users are defined as partners of the EHR im- 
plementation process and use or are expected to use 
EHRs in their everyday activities. A very recent Canadian 
study conclude that relatively few differences in percep- 
tions about EHR system adoption and use exist between 
physicians already using such systems and those not yet 
using the systems [15]. A recent qualitative study among 
key stakeholders from various groups in Canada (provin- 
cial and regional representatives, health care profes- 
sionals, public health agents, and vendors) reports 
important achievements and barriers to EHR implemen- 
tation, but this study does not differentiate the views be- 
tween these stakeholder groups [16]. This paper 
presents the findings of a Delphi study which explored 
the perspectives of different actual or potential EHR 
users in Canada. Achieving consensus among users on 
factors influencing the successful implementation of 
EHR through the Delphi technique appears as an im- 
portant first step given the current lack of empirical evi- 
dence to inform decision-making on possible effective 
implementation strategies in the Canadian healthcare 
system. These preliminary data could be used as the 
basis for empirical study of effective EHR implementa- 
tion strategies. 

The Delphi technique 

The Delphi technique is based on a structured process 
for collecting and distilling knowledge from a group of 
experts by means of a series of questionnaires inter- 
spersed with controlled opinion feedback [17-19]. It is 
considered to be a strong methodology for achieving a 
rigorous consensus of experts on a specific theme. There 
is no consensus on the panel size for Delphi studies. As 
shown by Akins, Tolson & Cole (2005), reliable out- 
comes could be obtained with a relatively small Delphi 
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panel of experts with similar training and general under- 
standing in the field of interest [20], However 
generalizability of the results may consequently be 
reduced. Although some degree of interpretation and 
flexibility have been observed, a classic Delphi survey 
follows a set of procedures reflecting behavioural and 
statistical processes [19]. Three rounds of questionnaires 
are typically sent to a chosen panel of experts. However, 
the number of rounds may vary. 

The main advantage of the Delphi technique is the 
achievement of consensus in a given area which is un- 
certain or lacks empirical evidence. This type of study is 
recommended for obtaining opinions from experts who 
live and work in different geographic regions and set- 
tings [19]. The feedback between rounds is interesting in 
itself given that it can be highly motivating and educa- 
tional for the participants. Finally, the anonymity of the 
Delphi technique also encourages open and honest feed- 
back among experts [19]. 

Findings of the systematic review 

The systematic review, covering a period from 1999 to 
2009, was conducted on nine electronic databases 
(PubMed, EMBASE, CINAHL, Business Source Premier, 
Science Citation Index, Social Sciences Citation Index, 
Cochrane Library, ABI/Inform, and PsychlNFO). The 
search strategy, developed by an information specialist, 
is available upon request. The key question was: "What 
can limit or contribute to the success of EHR implemen- 
tation projects according to each group of EHR users?". 
Studies were included if they reported on users' per- 
ceived barriers and facilitators to EHR implementation, 
in healthcare settings comparable to Canada. Studies 
with an empirical study design were included [21]. The 
systematic review identified four key EHR user groups: 
physicians, other non-physician health professionals, 
healthcare managers, and patients. The findings show 
that the most frequent implementation factors common 
to all user groups were: design and technical concerns, 
ease of use, interoperability, privacy and security, costs, 
productivity, familiarity and ability with EHR, motivation 
to use EHR, patient and health professional interaction, 
and workload and lack of time. Each user group also 
identified factors specific to their professional and indi- 
vidual priorities. More details can be found in the article 
presenting this study [21]. 

Study objectives 

This Delphi study aimed to assess the applicability, the 
importance, and the priority of the results of the system- 
atic review of international literature concerning users' 
perspectives of the factors influencing EHR implementa- 
tion [21,22]. It was driven by the desire to answer this 
question: What findings of the systematic review are 



applicable to the Canadian context and which factors are 
prioritized by each user group for future decision mak- 
ing regarding EHR implementation in Canada? 

Methods 

We conducted a Delphi study among Canadian repre- 
sentatives of actual or potential EHR users to confirm 
the findings of the systematic review and to prioritize 
the key barriers and facilitating factors for EHR imple- 
mentation in Canada. 

Participants 

The study participants are Canadian representatives of 
actual or potential EHR users, or groups who use or 
are expected to use EHR. They are subdivided into 
four groups: physicians, patients, managers and non- 
physician health [23-25]. In order to better reflect the 
reality of the Canadian healthcare system, we subdi- 
vided non-physician health professionals in two groups: 
non-physician healthcare professionals and health in- 
formation professionals. Non-physician healthcare pro- 
fessionals are nurses, physician assistants and other 
non-physician providers. Health information profes- 
sionals (HIP) are defined as people providing leader- 
ship in all aspects of clinical information management 
at both the micro and macro levels. At the micro (or 
individual record level), HIP professionals support data 
collection, use, access and disclosure, to the retention 
and destruction of health information regardless of for- 
mat. HIPs perform qualitative analysis on the docu- 
mentation within the health record and are responsible 
for the security of health records. HIPs are advocates 
of the individual's right to private, secure and confi- 
dential health information. At the macro (or aggregate 
data level), HIPs deal with the information through the 
health system, analyze statistics, manage complex 
information systems including registries and work with 
public, private and key stakeholders in understanding 
and using health data to improve the health of 
Canadians [26]. 

A purposive sampling method was used to recruit par- 
ticipants from all Canadian provinces and territories. 
More than 20 national and provincial e-health and 
healthcare professional associations, organizations and 
interest groups, such as the Canada Health Infoway 
Clinician Peer Support Network, were asked to collabor- 
ate through forwarding our study recruitment email to 
their members (see the full list in Additional file 1). 
Experts who were known actual or potential users of 
Canadian EHR systems were purposefully selected and 
also invited by telephone and email to join the study. 
Other potential key informants were identified through 
a snowball sampling technique. Interested participants 
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were directed to the study website to register their con- 
tact information and detail their EHR experience. 

Our selection criteria aimed to ensure an adequate 
breadth of expertise and representation of various geo- 
graphical contexts. Eligible participants were required to 
have professional experience related to Canadian EHRs. 
Because there are few fully interoperable EHR projects 



in Canada, "EHR experience" was broadly defined as 
being knowledgeable about the electronic organization 
of patient information, including a full or partial compu- 
terized record of a person's medical history, or just cer- 
tain records, such as laboratory or diagnostic testing 
results. Participants also needed to belong to one of the 
five EHR user groups (physicians, non-physician 



Table 1 Physician consensus 



Strength of 


EHR implementation factor 


Criteria* 


Consensus* 


5-point 


Percentile score 


Inter- 


consensus 




A — annlirahtilitx/ 

AA — OUUIItwCIUII 1 ly 


/ AnrPPinpnt 
\r\ui ccl I ICI lli 


Likert 


10th 


25th 


m la ft\ le> 

UJUOI IMC 






1 = importance 


in %) 


score 


range 


Strong 


Confidence in EHR developer or vendor 


A 


> 75 % 


4 


2 


4 


0 


1 


> 75 % 


4 


4 


4 


0 


Moderate 


Cost issues (start-up and maintenance) 


A 


> 75 % 


4 


3 


4 


0 


1 


60-74 % 


4 


3 


4 


0 


Lack of time and workload (clinical tasks) 


A 


60-74 % 


4 


3 


4 


0 


1 


60-74 % 


4 


4 


4 


1 


Partial 


Cost issues (return on investment) 


A 


60-74 % 


4 


I 


2 


2 


1 


< 60 % 


- 


1 


3 


1 


Patient and health professional interaction 


A 


< 60 % 


- 


3 


3 


1 


1 


60-74 % 


4 


3 


3 


1 


Lack of time and workload (EHR use) 


A 


60-74 % 


4 


2 


4 


0 


1 


< 60 % 


- 


4 


4 


1 


Change in tasks 


A 


60-74 % 


4 


3 


4 


0 


1 


< 60 % 


- 


3 


4 


1 


Choice of the EHR system 


A 


60-74 % 


4 


2 


3 


1 


1 


< 60 % 


— 


2 


3 


1 


None 


Design and technical concerns 


A 


< 60 % 


- 


3 


3 


2 


1 


< 60 % 


- 


3 


3 


2 


Privacy and security concerns 
(patient privacy) 


A 


< 60 % 


- 


1 


2 


2 


1 


< 60 % 


- 


1 


2 


2 


Privacy and security concerns 
(professional confidentiality) 


A 


< 60% 




1 


1 


3 


1 


< 60 % 




1 


3 


2 


Quality standards 


A 


< 60 % 




3 


3 


2 


1 


< 60 % 




3 


3 


2 


Productivity (loss of) 


A 


< 60 % 




3 


3 


2 


1 


< 60% 




3 


4 


1 


Practice size (small) 


A 


< 60 % 




2 


2 


2 


1 


< 60 % 




2 


3 


2 


Practice size (large) 


A 


< 60 % 




3 


3 


2 


1 


< 60 % 




3 


3 


2 


Physician salary status and reimbursement 


A 


< 60 % 




2 


2 


2 


1 


< 60 % 




2 


3 


1 


Human resources (IT support, other) 


A 


< 60 % 




1 


3 


2 


1 


< 60 % 




3 


4 


1 


Management 


A 


< 60 % 




1 


2 


1 


1 


< 60 % 




2 


2 


1 



§ A = applicability, I = importance. 

* Agreement in %: > 75 % agreement, 60-74 % agreement, < 60 % agreement. 
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healthcare professionals, health information profes- 
sionals, healthcare managers and patients), have a valid 
email address and access to the internet, and speak Eng- 
lish or French. As required by the ethics committee, pa- 
tient participants were also required to hold an official 
position of "patient representative" within a health asso- 
ciation or organization. 

Ethical consideration 

The Delphi study participants were given specific con- 
sent forms presenting research objectives and informa- 
tion about research implications. They were informed 
that their participation in the study was entirely volun- 
tary and that they implicitly consented to participate by 
completing the first round of the electronic Delphi 
study. Ethics approval for the study protocol was 
received from the Research Ethics Board of the Centre 
Hospitalier Universitaire de Quebec (approved January 
23, 2009; ethics number 5-08-12-06). 



Design of the Delphi study 

Based on factors identified for each users group in the 
systematic review [21] we designed a specific question- 
naire for each actual or potential user group. The ques- 
tionnaire proposed a number of questions that 
corresponded to the most frequent factors found for each 
user group in the literature review [21]. In order to limit 
the length of questionnaires we decided to select the fac- 
tors that were mentioned by at least three studies, or ex- 
ceptionally by two studies if one of them was conducted 
in Canada. Most studies included in the systematic re- 
view did not make a distinction between non-physician 
healthcare professionals and health information profes- 
sionals. Consequently, it was then not possible to extract 
the factors specific to each non-physician health profes- 
sional group. Healthcare professionals and health infor- 
mation professionals received the same questionnaire 
version (14 factors listed in Additional file 2), but each 
group was analysed separately because their answers 
were different. Then, four bilingual (English and French) 



129 

Individuals registered on study 
website to participate 



106 

Individuals invited to participate 



23 individuals not invited to participate 

8 refused because of inadequate or no EHR 
experience 

1 1 refused because not representing target user 
group 

2 could not participate because no patient Delphi 
group 

2 withdrew from project before invitation to 
participate sent 



Delphi study 
user group 


Physicians 


Healthcare 


Managers 


Health 
information 
professionals 




• 


■ 












Invited 
n (%) 


14( 


-) 


30(-) 


33(-) 


29(-) 








1 




Round 1 
n (%) 


10 (71.4) 


24 (80.0) 


25 (75.8) 


24 (82.8) 



Round 2 
n (%) 


7 (50.0) 


20 (66.7) 


20 (60.6) 


22 (75.9) 






J 






Round 3 

n(%) 


7 (50.0) 


17 (56.7) 


18 (54.5) 


22 (75.9) 



Figure 1 Flow diagram of the Delphi study participants. 
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online questionnaire versions, pertaining to each user 
group (physicians, non-physician health professionals, 
managers and patients) were developed (see complete 
questionnaires presented in Additional files 3, 2, 4, and 
5). Two bilingual members of the research team validated 
the translation. The questionnaires were pre-tested by 
members of the research team to assess the clarity of the 
questions, the instructions and the format. Physicians 
and managers received questionnaire versions with re- 
spectively, 18 and 11 factors. Using a five-point Likert 
scale (where 1 indicated strong disagreement and 5, 
strong agreement), participants rated each questionnaire 
statement for its applicability to the participant's specific 
EHR implementation context and its importance for 
decision-making regarding EHR implementation in 
Canada. Participants were also invited to leave written 
comments after each answer and general comments at 
the end of the questionnaire. The Delphi study was con- 
ducted during a 10 day period in March 2010, using a 
dedicated website developed for the purpose of this 
study. 

For each Delphi study round, participants were 
emailed a link to the appropriate questionnaire on the 
project website, and were allotted 48 hours to complete 
the questionnaire. Email reminders to complete the 
questionnaire were sent after 24 hours to participants 
who had not yet replied. In the first round, participants 
were asked to simply rate their responses for each item. 
In the second and third rounds, distributions of partici- 
pants' answers to each item in the previous round were 
presented in percentage form. Participants were invited 
to take into consideration the other participants' 
responses and reassess their answers in light of this new 
information. Questionnaire formats remained un- 
changed during the three study rounds and participants 
were not provided with reminders of their responses in 
the previous rounds. Individuals who did not complete a 
previous round were not invited to participate in follow- 
ing rounds. 

Analysis 

Third round data was used for final analysis. As there is 
no existing definite criteria determining consensus in a 
Delphi study [27], we chose a priori consensus criteria 
based upon the research team's previous work [28]. Con- 
sensus on a questionnaire factor was considered "strong" 
when at least 75% of participants reached an agreement 
on both the applicability and importance. We chose to 
consider these two criteria because this would indicate a 
priority for decision making [18]. "Moderate" consensus 
required 60% to 74% of participants to agree on both the 
applicability and the importance. "Partial" consensus was 
obtained when at least 60% of the participants reached 
consensus on only one aspect (applicability or 



importance) of a factor. Absence of consensus was deter- 
mined when less than 60% of participants agreed on the 
applicability and the importance of a factor. 

Percentile scores and interquartile range were used 
to calculate the level and strength of the consensus, 
respectively. To determine the level of consensus, 
tenth and twenty-fifth percentile scores were calcu- 
lated. Tenth percentile scores indicate the lowest num- 
ber on the Likert scale upon which at least 90% of 
participants agreed and 25th percentile scores indicate 
75% agreement. Interquartile range, a measure of stat- 
istical dispersion, indicated the strength of the consen- 
sus, where 0 specifies a strong group consensus and 2 
indicates dispersed responses. For example, in Table 1 
physician consensus, the factor "confidence in EHR 
developer or vendor" (applicability) shows that accord- 
ing to 10 th percentile scores 90% of respondents 
responded either 2, 3,4 or 5 on the 5 point Likert scale, 
while 75% of respondents responded either 4 or 5. The 
interquartile range of 0 indicates strong group 
consensus. 

Priority for decision-making concerning EHR imple- 
mentation in Canada was determined based on the ques- 
tionnaire items upon which 90% of participants 
considered both applicable and important (that is, fac- 
tors scoring 4 or 5 for applicability and importance on 
the 5-point Likert scale). Tenth percentile scores were 
used to determine if the questionnaire item reached 



Table 2 Participant demographics, round 3 (n = 64) 



Characteristic 




n (%) 


Profession 




Health information professional 


22 (34.4) 




Healthcare manager 


18 (28.1) 




Nurse 


1 5 (23.4) 




Occupational therapist 


1 (1.6) 




Pharmacist 


1 (1.6) 




Physician 


7 (10.9) 


Sex 




Female 


52 (81.2) 




Male 


12 (18.8) 


Province or territory of residence 




Alberta 


12 (18.8) 




British Columbia 


6 (9.4) 




Manitoba 


2(3.1) 




Newfoundland 


1 (1.6) 




Nova Scotia 


4 (6.2) 




Ontario 


28 (43.8) 




Quebec 


7 (10.9) 




Saskatchewan 


2(3.1) 




Yukon 


2(3.1) 
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sufficient consensus. The analyses were performed using 
SAS software version 9.1. 



in their professional occupations and area of residence 
(Table 2). 



Results 

Characteristics of the participants 

Among 106 registered individuals invited to join one of 
the four Delphi study groups, 64 participants responded 
to all three rounds (Figure 1). The response rates were 
satisfactory and varied between 50% and 76% across the 
Delphi groups in the third round. We recruited a suffi- 
cient number of participants from the physician, other 
non-physician healthcare professionals, health informa- 
tion professionals, and managers groups, but not for 
patients (only two patient representatives completed the 
questionnaire, and this group was thus excluded). 
Respondents were mostly female (81%) and varied both 



Results of the Delphi study 

Table 1 shows that nearly half (8/18) of the factors on 
the physicians' questionnaire reached consensus on the 
applicability or importance: one item reached a strong 
consensus (> 75%), two items moderate consensus 
(> 60-74%), and five items partial consensus (> 60%). 
Healthcare professionals reached consensus on all fac- 
tors of their questionnaire (Table 3). Health information 
professionals attained consensus on all but one of the 14 
factors to a strong, moderate and partial level on 5, 4 
and 4 factors, respectively (Table 4). Out of a total of 11 
factors, managers achieved strong, moderate, and partial 
consensus on 2, 2, and 3 factors, respectively (Table 5). 



Table 3 Healthcare professional consensus 



Strength of 


EHR implementation factor 


Criteria 5, 


Consensus* 


5-point 


Percentile score 


Inter- 


consensus 




A = applicability 


(Agreement 


Likert 


1 0th 


25th 


quartile 






1 = importance 


in %) 


score 


range 


Strong 


Perceived usefulness 


A 


> 75 % 


5 


4 


5 


0 


1 


> 75 % 


5 


5 


5 


0 


Motivation 


A 


> 75 % 


5 


4 


5 


0 


1 


> 75 % 


5 


4 


5 


0 


Patient and health professional interaction 


A 


> 75 % 


4 


4 


4 


0 


1 


> 75 % 


4 


4 


4 


0 


Lack of time and workload 
(professional tasks) 


A 


> 75 % 


5 


4 


5 


0 


1 


> 75 % 


5 


5 


5 


0 


Lack of time and workload (EHR use) 


A 


> 75 % 


5 


5 


5 


0 


1 


> 75 % 


5 


5 


5 


0 


Resources available (additional) 


A 


> 75 % 


5 


4 


5 


0 


1 


> 75 % 


5 


5 


5 


0 


Human resources (IT support, other) 


A 


> 75 % 


5 


3 


5 


0 


1 


> 75 % 


5 


3 


5 


0 


Participation of end-users in 
implementation strategy 


A 


> 75 % 


5 


4 


5 


0 


1 


> 75 % 


5 


5 


5 


0 


Moderate 


Productivity 


A 


60-74 % 


5 


4 


4 


1 


1 


> 75 % 


5 


4 


5 


0 


Partial 


Design and technical concerns 


A 


< 60 % 




4 


4 


1 


1 


> 75 % 


5 


4 


5 


0 


Perceived ease of use 


A 


< 60 % 




4 


4 


1 


1 


> 75 % 




4 


5 


0 


Privacy and security concerns 


A 


> 75 % 


3 


2 


3 


0 


1 


< 60 % 




3 


3 


2 


Outcome expectancy 


A 


< 60 % 




4 


4 


1 


1 


> 75 % 


5 


4 


5 


0 


Management 


A 


< 60 % 




4 


4 


1 


1 


> 75 % 


5 


4 


5 


0 



§ A = applicability, I = importance. 

* Agreement in %: > 75 % agreement, 60-74 % agreement, < 60 % agreement. 
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Table 4 Health information professional consensus 



Strength of 


EHR implementation factor 


Criteria 8 


Consensus* 


5-point 


Percentile score 


Inter- 


consensus 




A = applicability 
1 = importance 


(Agreement 
in %) 


Likert 
score 


10th 


25th 


quartile 
range 


Strong 


Perceived usefulness 


A 


> 75 % 


5 


4 


5 


0 






1 


> 75 % 


5 


4 


5 


0 




Productivity/efficiency 


A 


> 75 % 


5 


4 


5 


0 






1 


> 75 % 


5 


5 


5 


0 




Motivation 


A 


> 75 % 


5 


4 


5 


0 






1 


> 75 % 


5 


4 


5 


0 




Management 


A 


> 75 % 


5 


5 


5 


0 






1 


> 75 % 


5 


5 


5 


0 




Participation of end-users in 


A 


> 75 % 


5 


5 


5 


0 




implementation strategy 


1 


> 75 % 


5 


5 


5 


0 


Moderate 


Design and technical concerns 


A 


60-74 % 


5 


3 


4 


1 








60-74 % 


5 


A 


4 


1 




Perceived ease of use 


A 


60-74 % 


5 


3 


4 


1 






1 


60-74 % 


5 


A 


4 


1 




Resources available 


A 


> 75 % 


5 


4 


5 


0 








60-74 % 


5 


3 


4 


1 




Human resources (IT support, other) 


A 


60-74 % 


5 


3 


4 


1 








> 75 % 


5 


4 


5 


0 


Partial 


Privacy and security concerns 


A 


< 60 % 




3 


3 


2 








60-74 % 


5 


4 


4 


1 




Outcome expectancy 


A 

A 


< DU 70 




4 


4 








1 


60-74 % 


5 


4 


4 






Lack of time and workload 


A 


< 60 % 




4 


4 






(professional tasks) 


1 


60-74 % 


5 


2 


4 






Lack of time and workload (EHR use) 


A 


< 60 % 




2 


3 








1 


> 75 % 


5 


3 


5 


0 


None 


Patient and health professiona 


A 


< 60 % 




2 


2 


2 




interaction 


1 


< 60 % 




2 


3 


2 



s A = applicability, I = importance. 

* Agreement in %: > 75 % agreement, 60-74 % agreement, < 60 % agreement. 



Comparison between all groups is limited, as each 
group received a different questionnaire adapted to re- 
flect the findings of the systematic review [21] specific to 
each actual or potential user group. Figure 2 shows the 
interrelationship of questionnaire items. As health pro- 
fessionals and health information professionals received 
the same questionnaire, they are most easily compared 
and generally shared points of view for most question- 
naire items. The three following factors were common 
to all questionnaires, though question format varied: 
"privacy and security concerns", "lack of time and work- 
load", and "human resources". Consensus for these three 
factors varied among all groups (Table 6). 

As indicated in the Additional files 3, 2, 4, and 5, each 
of the items was associated to an EHR implementation 
factor. Ten decision-making factors, specific to EHR im- 
plementation in Canada (Table 7), are among the 



elements of strong consensus for health professionals, 
health information professionals, and managers. These 
10 factors are perceived usefulness, productivity, motiv- 
ation, participation of end-users in the implementation 
strategy, patient and health professional interaction, lack 
of time and workload, resources availability, manage- 
ment, outcome expectancy, and interoperability. These 
factors were prioritized on the account that at least 90% 
of participants agreed with a Likert scale rating of 4 or 
5. Because this criterion was not satisfied, we were un- 
able to prioritize physicians' responses. 

Discussion 

This study aimed to gather this first-hand knowledge 
held by actual and potential Canadian EHR users. Using 
the Delphi method, we compared the implementation 
experiences in the Canadian context with the known 
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Table 5 Manager consensus 

Strength of EHR implementation Criteria 5 Consensus* 5-point Percentile score Inter-quartile 

consensus factor A = applicability (Agreement in %) Likert score 25th - ran 9 e 

I = importance 

Strong Interoperability A > 75 % 5 5 5 0 

I > 75 % 5 5 5 0 

Outcome expectancy A > 75 % 5 5 5 0 

I > 75 % 5 5 5 0 

Moderate Resources available A 60-74 % 5 14 1 

I > 75 % 5 4 5 0 

Training A 60-74% 5 13 2 

I > 75 % 5 4 5 0 

Partial Cost issues A < 60 % - 2 4 1 

I > 75 % 5 4 5 0 

Human resources A < 60 % 14 1 

(IT support, other) — j~ > 75 % ~~ o~ 

Choice of the system A < 60 % 2 3 2 

I 60-74% 5 2 4 1 

None Privacy and security concerns A < 60 % 13 1 

(security of patient information) j <~60~% - 2 3 1 

Privacy and security concerns A < 60 % - 2 2 2 

(patient privacy) I < 60 % - 2 4 1 

Familiarity, ability with EHR A < 60 % 11 1 

I < 60 % 11 1 

Lack of time and workload A < 60 % 12 1 

I < 60 % - 1 2 3 

§ A = applicability, I = importance. 

* Agreement in %: > 75 % agreement, 60-74 % agreement, < 60 % agreement. 



factors found in the scientific literature. Our findings 
show that, overall, many of the factors found in the sci- 
entific literature are also applicable to the Canadian con- 
text. However the varying levels of agreement between 
and within user groups could mean that each group has 
unique professional priorities in the EHR implementa- 
tion process. 

This study is addressing an issue of great relevance in 
the current Canadian health care context. Notably, key 
factors to EHR implementation were considered of the 
highest priority in a recently published report investigat- 
ing gaps in knowledge, research, and research capacity 
regarding EHRs in the Canadian primary health care 
[29]. 

However, the results of this study must be interpreted 
cautiously due to some limitations. In fact, the major 
flaw in this kind of study is that it relies on the percep- 
tion of participant on what might potentially influence 
implementation success without actual evidence from 
real implementation. We would like to invite EHR im- 
plementation researchers to further explore this list of 
priorities in order to validate them in diverse implemen- 
tation contexts. Also, despite exceptional recruitment 



efforts, only two representatives of patient groups volun- 
teered to take part in the study. Consequently, the pa- 
tient Delphi study was cancelled creating an important 
gap to our findings. As mentioned above, the small sur- 
vey sample may reduce generalizability of the results. 
Also, the 50% participation rate of physicians caused 
limited prioritization from this group. As well, a bias 
introduced by a participation dominated by women 
could occur. Although profession is an important factor 
contributing to group consensus, other factors such as 
age, work setting, and specific type of EHR experience 
are influences that were not considered in this Delphi 
study. Consequently, the factors presented might have 
not been exhaustive of the respondents' experiences and 
important factors may have been overlooked. Secondly, 
we do not know if the participants that dropped out of 
the study and the resulting missing data could represent 
a bias for the degree of consensus. Finally, results of this 
study are based upon the informed opinions of the parti- 
cipants, and results should be interpreted appropriately 
as the well-founding of such opinions cannot be verified. 

The three factors common to all user groups reached 
different levels of consensus (see Table 6). Firstly, 
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Physicians 

■ Confidence in EHR developer or 
vendor 
- Change in tasks 
- Quality standards 
- Productivity (loss of) 
- Practice size 
- Physicians salary status and 
reimbursement 




- Management 

• Patient and Health 

professional 

interaction 

- Design and 

"'^technical concerns., 
-- 



\ 
\ 
l 
I 

y 



- Cost issues 
-Choice of the EHR 

system 



- Privacy and security 

concerns 
• Lack of time and workload 

- Human resources (IT 
support and other) 



Healthcare managers 



- Interoperability 
-Training 
■ Familiarity, ability with EHR 




I 



Specific factors 



Healthcare professional and 
information professional 

- Perceived usefulness 
- Motivation 
- Participation of end-users in 
implementation strategy 

- Productivity / efficiency 

- Perceived ease of user 
- Management 




( \ Common factors 

^ 

Figure 2 Common and specific EHR adoption factors among Delphi groups. 



physicians and managers disagreed while health profes- 
sionals and health information professionals partially 
agreed on the applicability or the importance of "Privacy 
and security concerns". Our research question was cen- 
tered on "which" factors were a priority. The limited con- 
sensus reached on certain aspects of the EHR 
implementation raises the question of "why" this is. 
Range in practice size or differences in professional set- 
tings could be an explanation in some instances, however 
only a dedicated study can answer this question. Sec- 
ondly, perceptions toward "Lack of time and workload" 
appear to be user specific as indicated by strong, 
moderate, partial or no consensus for health profes- 
sionals, physicians, health information professionals, and 
managers, respectively. Thirdly, "Human resources" pre- 
sents a different consensus pattern in which health pro- 
fessional reached a strong consensus whereas health 
information professionals reached a moderate consensus. 
Additionally, physicians disagreed while managers par- 
tially agreed on the applicability or the importance of 
"Human resources". Consensual factors of applicability 
and importance identified in our study for each user 
group should guide future research. We compared the 
Delphi study results with the factors specifically 



mentioned by the Canadian studies included in the sys- 
tematic review [30-35]. Firstly, among the 17 studies 
included in our systematic review pertaining to physi- 
cians, the factor "confidence in the EHR developer" was 
mentioned only twice (including one Canadian study 
[33]). However, Canadian physicians reached a strong 
consensus about this factor in our Delphi study, indicat- 
ing that this is an issue of particular relevance that 
should be explored in further research. Furthermore, 
while one Canadian study identified fee-for-service pay- 
ment of Canadian physicians as a barrier to EHR imple- 
mentation [33], our physician participants did not reach 
consensus on this subject. Updated research by the same 
authors confirms our findings, suggesting that this remu- 
neration approach does not hinder EHR implementation 
more than other forms of payment [36]. Our findings 
concerning healthcare professionals are congruent with 
those of a recently published Canadian case study [37]. 
This study identified facilitating EHR implementation 
factors such as perceived usefulness, motivation, effective 
onsite technical support and management, and barriers 
such as concerns about privacy and security. Participants 
in our Delphi study reached consensus on all these fac- 
tors. Moreover, while no Canadian study included in our 
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Table 6 Comparison of similar EHR implementation factors among Delphi groups 



EHR implementation factor 




User group 8 (strength of consensus*) 






P 


HP 


HI 


M 


Privacy and security concerns 


None 


Partial 


Partial 


None 


Lack of time and workload 


Moderate/Partial 


Strong 


Partial 


None 


Human resources (IT support, other) 


None 


Strong 


Moderate 


Partial 



5 P = physicians, HP = healthcare professionals, HI = health information professionals, M = managers. 

* Strong = > 75 % of participants agree on the applicability and importance EHR implementation factor, Moderate = > 60-74 % of participants agree on the 
applicability and importance EHR implementation factor, Partial = > 60 % of participants agree on one aspect of an EHR implementation factor, either its 
applicability or importance, None = < 60 % of participants agree on the applicability and importance of an EHR implementation factor. 



systematic review mentioned the factors "lack of time 
and workload" or "resources available", we found that 
Canadian healthcare professionals perceive these factors 
as important barriers to EHR implementation. A recent 
Canadian report [38] confirms our findings, indicating 
that significant labour and skills shortages among health 
professionals experienced with electronic health informa- 
tion systems are likely to constrain the successful imple- 
mentation of such systems in Canada. 

Conclusion 

This study provides some key findings that corroborate 
and expand current knowledge of the factors influencing 
EHR implementation in Canada. We conclude that the 
perspectives and priorities concerning the barriers and 
facilitating factors to EHR implementation in Canada 
could vary greatly between and within user groups and 
present a challenge to successful implementation of 
EHR programs [39]. 

In summary, we would like to invite EHR implementa- 
tion researchers to further explore the list of priority fac- 
tors specific to Canadian user groups that were 
identified in this Delphi study. Although important re- 
cruitment efforts have been made, it was not possible to 
recruit sufficient patient representatives and the patient 



Delphi study was thus cancelled. This limitation should 
be taken into account in future research. 



Summary table 

What was already known on this subject 

There is currently a strong focus worldwide on the 
potential of EHRs to reduce healthcare costs and 
significantly improve the quality of healthcare provided. 
The implementation of EHR programs is complex and 
involves many user groups (e.g. physicians, healthcare 
professionals, health information professionals, 
managers, and patients). 
Understanding users' perspectives of EHR 
implementation is essential to the effective 
implementation of EHRs and to the successful 
integration of EHRs into the healthcare system. 

What this study added to our knowledge 

Amongst all barriers and facilitators to EHR 
implementation identified in a previous systematic 
review, ten were prioritized in the context of the 
Canadian healthcare system through this Delphi 
study. 



Table 7 Prioritization of EHR implementation factors for decision-making, by EHR user group 







User group* 




EHR implementation factor 


Healthcare professionals 
(Agreement in %) 


Health information professionals 
(Agreement in %) 


Managers 
(Agreement in %) 


Perceived usefulness 


> 75 % 


> 75 % 




Productivity 


> 75 % 


> 75 % 




Motivation 


> 75 % 


> 75 % 




Patient and health professional interaction 


> 75 % 






Lack of time and workload (professional 
tasks & EHR use) 


> 75 % 






Resources available 


> 75 % 






Participation of end-users in implementation 


> 75 % 


> 75 % 




Management 




> 75 % 




Outcome expectancy 






> 75 % 


nteroperability 






> 75 % 



*Physicians are not presented in this table as this user group did not meet prioritization criteria. 
* Agreement in %: > 75 % agreement. 
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Users' perspectives of barriers and facilitators are 
complex and each user group has unique 
professional priorities and roles in the EHR 
implementation process. 

We recommend that decision-makers integrate the 
perspectives specific to each user group, and con- 
sider the elements of consensus that emerged from 
this Delphi study, when implementing future EHR 
projects. 

Additional files 



Additional file 1: Associations, organisations, and interest groups 
solicited to participate in the Delphi study. 

Additional file 2: Healthcare professional and health information 
professional questionnaire. 

Additional file 3: Physician questionnaire. 

Additional file 4: Manager questionnaire. 

Additional file 5: Patient questionnaire. 
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